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FOREWORD 



The use of indicators has proved invaluable in the 
business sector for many years. Likewise, social indicators 
have been utilized widely to assist in the determination of 
future needs relative to varied aspects of American life. 

Now indicators of performance have entered the field of 
education. The State Department of Education in Oregon has 
been a leader in the gathering and application of educational 
indicators. The department's publication, indicators and 
statewide Assessment, published by the Cooperative Account- 
ability Project (CAP) , is a valuable addition to the rcipidly 
growing collection of assessment and accountability litera- 
ture. CAP also will publish another indicator-related 
document based on the work of Thomas E. Kendig and J. R-i rt 
Coldiron in the Pennsylvania Department of Education. T; • y 
are examining the indicators identified by Oregon and 
establishing relationships to specific educational condi- 
tion variables which have been collected in Pennsylvania. 

Additionally CAP asked Michael J. Grady, Jr. to prepare 
this monograph as a guide to the use of indicators by school 
district personnel who are concerned with evaluating the 
effectivenet'o of their school programs. Dr. Grady has given 
full consideration to the Oregon publication noted above and 
to the work that is being done in Pennsylvania preparatory 
to that state's publication. His intent is to build a bridge, 
as it were, between the two documents and to provide specific 
guidelines for the application of indicators at the local 
school district level. 

It should be noted that state departments of education 
across the country have been the intermediaries between 
legislators and state boards of education on the one hand and 
local school districts on the other. For the past several 
years state department personnel have been talking about new 
responsibilities in assisting school districts with their 
educational accountability efforts, but the progress in this 
direction has been slow. Perhaps this is because too few 
people were giving serious thought to appropriate evaluation 
procedures at either the local or state levels before the 
accountability movement appeared on the scene. Since the 
basic responsibility for education in this country rests with 
each state, it is important to recognize that state departments 
of education are in a favorable position to provide the needed 
leadership and assistance to improve evaluation processes 
throughout their jurisdictions. 



while school districts long have been expected to be 
accountable for the implementation of educational lavs and 
state school board regulations concerning such matters as 
transportation, requirements for school facilities, lunch 
programs, and so forth, they now are being asked to be 
accountable for the performance of their students. Educators 
widely recognize that the traditional written test — by itself 
— cannot provide a true and complete evaluation of student 
outcomes. Now the introduction of indicators to the educa- 
tional field may help to give greater impetus to broad 
evaluation efforts so necessary at both state and local levels. 

With the increasing commitment of student time, school 
personnel, and public money to the educational enterprise, 
many questions continue to be raised about school system 
operction. Since quality education has become a public by- 
word and educational accountability is being stressed on all 
fronts — local, state, and national — the question is being 
asked, "Do educators have specific evidence upon which the 
quality of the educational program can be judged?" This mono- 
graph on Using Educational Indicators for Program Accountability 
will help to provide procedures for answering that question. 
Providing evidence is the responsibility of local, state, and 
national agencies. Judgment based on the evidence rests with 
the various publics concerned with education. Each must make 
sound decisions within its area of responsibility. These 
publics include legislators, taxpayers, students, local school 
administrators, and members of state departments of education. 
Accountability will work if each of these publics accepts its 
responsibility for decision making based on the evidence in 
the form of indicators of performance. 

I sincerely hope this monograph will assist educators — 
as well as legislators and concerned laymen — in the search for 
readily available and relatively inexpensive indicators of 
performance upon which decisions can be made concerning the 
effectiveness of educational programs. 



Arthur R. Olson, Director 
Cooperative Accountability Project 
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CHAPTER I 



INTRODUCTION 



Traditionally one of the unique features of American 
public education has been the meaningful involvement of local 
citizenry in providing support for the financing, adminis- 
tration, and guidance of educational programs. Actual teach- 
ing and corresponding educational learning procedures are 
assigned to professional educators and supportive personnel 
employed by a given school district. Citizens provide the 
resources necessary to implement the educational programs 
which they have authorized through their representatives on 
the school board. Ideally educators then complete the cycle 
by supplying citizens with information intended to show how 
much students have attained as a result of the community's 
concerted efforts. 

Educators' reports to local citizens concerning student 
learning progress have tended to focus on achievement test 
information. Generalizations designed to reassure the local 
public have included statements such as: "Our students rank 
in the top 70% of stu'.^ents nationally in mathematics"; "Our 
high school students have achieved in the top 10% of all 
students in our state in social studies"; or "Our elementary 
students scored at the national average on reading speed and 
comprehension. " 

Such educator assurances usually were acceptable to 
local citizens until the mid-1950s. But when the Russians 
successfully fired their Sputnik, many Americans began asking, 
"Why wasn't America first in space?" One prominent response 
at the national level was that U.S. public education did not 
include sufficient mathematics or science to allow American 
learners to be on a parity with their Russian counterparts. 
This rationalization was seized upon by Admiral H. G. Rick- 
over who claimed that high school graduates were not capable 
of learning to perform work on nuclear submarines without 
C07 siderable additional training. Educational leaders across 
the country took issue with Admiral Rickover's thesis, and 
the national educational debate began. Educators voiced the 
argument that American public education was basically sound 
but, with additional educational counselors and financial 
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resources, it could be even better. Legislation quickly 
passed to provide federal funds for counselor training, 
audio-visual equipment, and improving existing mathe- 
matics and science courses. In 1965 these programs were 
enlarged through the passage of the Elementary and 
Secondary Education Act which made funds available for a 
wide variety of special learners, innovative educational 
practices, and the strengthening of educational leader- 
ship in each state. 

The American public readily accepted federal 
supplemental funding of local educational programs. It 
had much to recomiaend it. Students gained additional 
educational opportunities which did not have to be financed 
with local tax resources. Local educators continued their 
usual means of reporting student achievement information to 
the public, and the public seemed satisfied. 

But by the early 1970s economic and social upheavals 
and corresponding increases in taxes began to affect every 
aspect of American life. Now people started to ask, "What 
are we getting for all the money we're putting into educa- 
tion?" State legislators reacted quickly to this public 
concern and at this writing, over half of the 50 states have 
passed legislation on educational accountability , 

The nature of each state's educational accountability 
statutes varies from the mere reporting of achievement 
testing, pupil-teacher ratios, and dropout information to 
more extensive reporting by school districts including pro- 
gram cost information. Throughout educational accountability 
legislation, however, is the common concept which requires 
both responsibility and disclosure , 

Someone — or, more likely, many people — are responsible 
for the amount of learning each student attains. Certainly 
teachers, counselors, and educational administrators 
influence achievement. Students themselves have a responsi- 
bility for their own learning. Parents also play an active 
role in terms of their youngsters' learning. In addition 
leaders in business, commerce, industry, labor, and the 
trades, as well as the public at large, should be very con- 
cerned about the instructional process. They not only pay 
for the educational opportunities in their community, but 
also employ the products of public education. 

Now as we proceed through the 1970s, it no longer will 
be considered sufficient for educational accountability 
disclosure purposes to seek and report information concerning 



expenditure per students quantity of equipment and material 
resources, quality of human resources, student achievement 
on a school, state, regional, or national basis, or the 
percentage of graduates who enter college. Such data still 
are considered necessary, but it will not be sufficient for 
educational accountability purposes. 

The following questions, although not exhaustive, pro- 
vide an indication of the nature of additional information 
which may be required in the future if a school district is 
to be truly accountable to the public it serves: 

(The sample questions are directed to a sixth grade 
math program.) 

1. How much do the math materials cost in the Grade 6 
program? 

2. Cost/student (Cost per student) 

3. How much pupil achievement is attained for this 
cost? 

4. Are alternative commercially available math 
materials more cost effective? 

5. What is the anticipated usable life for the present 
math materials? 

6. What is the anticipated usable life for alternative 
math materials? 

7. Are alternative math materials available for gifted 
and slow learners as well as for the average stu- 
dent? Or are present materials comprehensive 
enough so these various student learning needs can 
be satisfied? 

8. Is the math curriculum for Grade 6 students rele- 
vant to their learning needs? 

9. Are the Grade 6 math materials racially biased? Do 
all ethnic youngsters achieve equally well with 
these materials? If not, what procedures have been 
employed to equalize the math educational opportu- 
nities for all students? 

10. Are the illustrations and probleras to be mastered 
by the student practical in terms of real life or 



career education applications of math theory, or 
are they theoretically oriented for the more 
academically talented or college-bound students? 

11. What percentage of students is functioning at or 
above grade level in Grade 6 math? 

12. re there certain minimum competencies in Grade 6 
.aath that are required of all students? Which ones? 
How many? How is mastery learning of these 
competencies achieved? 

13. Which Grade 6 math skills are most difficult for 
students to master? What has been done to lessen 
this problem? 

14. Are the district schools overcrowded? If so, what 
has been done to accommodate the overflow? Split 
sessions? Extended day? Modular scheduling? 
What impact has overcrowding had upon the student 
learning of Grade 6 math? What has been dene about 
this problflin? 



These questions (#1 - 14) could be applied to grades one 
through six programs in language arts, reading, or social 
studies. They also could be applied to grades seven through 
twelve English, social studies, science, physical education, 
and vocational education programs. 

The educational accountability process also may 
require the examination of additional questions of a more 
general nature, such a& the following: 

1. What is the dropout rate? Are dropouts proportion- 
ate to the ethnic enrollment of all students in the 
district? If not, what has been done about it? 



2. How does the cost per student vary from the 
elementary, junior high (or middle school) , and 
high school levels? Is this variance reasonable? 

3. What is the counselor-studert ratio at the high 
school level? At the junior high (or middle school) 
level? Are counselors operating in the elementary 
schools? 



4. Are counselors actually spending cheir time 

counseling students, or are they functioning as 
highly paid secretaries who deal only with student 
scheduling? 



5. Are comprehensive vocational courses available for 
all grades at the secondary level? If not, why not? 

6. Are teachers certificated, and do they have a 
collegiate background for all courses they teach? 
Are teacher salaries on a parity with similarly 
sized school districts rn the state? 

7. Do teachers, students, and citizens have an input 
into the specifications of the various curricula 
offered in the district? 

8. Does the district share its accountability find- 
ings with neighboring districts? If not, why not? 

9. Has the school district entered into cooperative 
research studies with other districts? If not, 
why should the district be duplicating expenditure 
of research funds being spent elsewhere in the 
state? 

10. Do the administrators suggest alternative educa- 
tional programs to replace those which are not 
presently cost effective in terms of locally 
available funds? 

Undoubtedly a few years will pass before many school 
districts can provide answers to the first group of fourteen 
sample questions on both grade and academic levels. Those 
questions dealing with costs probably will not be answerable 
until experimental studies on educational Program Planning- 
Budgeting-Evaluation Systems (PPBES) are completed, and 
workable procedures are available to ascertain needed cost 
data. Many answers relevant to the second set of ten 
questions, however, already are available at the local school 
district level. Granted, some time will be ueeded for dis- 
tricts to "tool up" to gather this information. The 
gathering process will necessarily involve the skillful use 
of educational indicators. 

Indicators 

Virtually every American adult uses indicators to assist 
in making decisions relative to purchasing a home (prime 
interest rate for mortgage money) , buying stocks (Dow- Jones 
averages) , judging the purchasing power of take-home pay 
(consumer price index) , etc. Likewise, some rtates (notably 
New York, Pennsylvania, and Oregon) collect information 
which provides evidence or indicators that their school 
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programs are doing what they are intended to do. For 
example, finding out how many college prep students are 
still in college one year after hie i school graduation and 
how many vocational students are gainfully employed one year 
after high school graduation in the vocation for which they 
prepared could go a long way to help answer the question, 
"How well do our college prep and vocational education pro- 
grams prepare students for college and the world of woik?" 
For both groups in the illustration, the local school 
district would have to develop a specific study to gather 
needed information. In all likelihood telephone interviews 
with parents or a questionnaire sent to graduates would have 
to be completed and the findings tabulated and evaluated. 

Other educational indicators might evolve from 
information already being collected either at the district, 
county, state, or federal levels. Essentially, these 
indicators would cost nothing to obtain. 

indicators can greatly assist local school adminis- 
trators — and the lay public — by allowing the existing 
educational program to be viewed in a different perspective 
from the traditional perspective. Indicators also can be 
employed to support the quality notion which an educational 
program had, has, or should have. 

The purpose of this monograph is to present a rationale 
for the use of educational indicators as significant compo- 
nents in the evaluation of school programs. .iS the reader 
will discover, indicators are not new; rather, it is 
suggested that the use of indicators by educators enables 
them to better evaluate the effectiveness of their programs. 

Generally school administrators will have four purposes 
in using educational indicators. These purposes can perhaps 
be best summarized in the following questions: 

1. How are our students doing in school? 

2. How do our students fare after they leave school? 

3. How can this information be of assistance to our 
instructional staff? 

4 r Are we getting as much as we should for our school 
dollar? 

These are four rather basic questions which trigger 
myriad sub-questions when related to a given educational 



program. For example, let's say that we want to focus on 
the sequence of commercial (business) courses being offered 
in our school. We could expand our basic questions some- 
thing like this: 

1. How are our students doing in school? Consider 

- grades assigned 

- commercial student morale 

- teacher-classroom observation 

2. How do our students fare after they leave school? 

- check with businessmen who have hired our graduates 

- note strengths as well as weak areas in our 
academic experience as related to whether or not 
commercial students are "making it" on the job 

- send questionnaires asking value of high school 
commercial courses for students enrolled in 
colleges or vocational schools 

3. How can this information be of assistance to our 
instructional staff? 

- appraise the staff of the information (indicator 
data) obtained from questions #1 and #2 

- the effectiveness of our business program in 
terms of graduate success on the job would tend 
to reinforce the key skill areas of our com- 
mercial course sequence 

- the weak areas identified would become areas 
for consideration of possible course revision 

4. Are we getting as much as we should for our school 
dollar? Let's say we have a four-course sequence 
in our commercial program. We should look at 

- cost per student for each of the four courses 
as well c»s the total cost of the jprogram 

- the problem ar^as discovered under question #2 
to see if some older, traditional material might 
be dropped in favor of including the problem area 
topics 
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- various cost comparisons from course to course to 
see if we could teach the skills required for the 
modern student in three or three and one-half 
courses instead of four... or if we really need 
four and one-half or five full courses in our 
commercial sequence 

When used in the manner just described, educational 
indicators can be of invaluable assistance to administrators 
teachers, school board members, and the public in helping 
our educational programs remain relevant, up-to-date, and 
cost effective for local purposes. 

The remainder of this monograph is devoted to an 
elaboration of the use of educational indicators either for 
accountability purposes or for planning and efficient manage 
ment. In Chapter II, indicators are defined more precisely, 
and the various kinds of indicators are described and 
illustrated. Chapter III is designed to answer the question 
"Where do you get indicators?" Chapter IV tells you how to 
collate indicator data for analysis. Chapter V suggests 
alternate ways for interpreting indicator data. Finally, 
Chapter VI presents suggestions concerning the dissemina- 
tion of indicator data. Throughout, the approach is one of 
"how- to-do- it" rather than the creation of a theoretical 
treatise on educational indicators. 



8 



CHAPTER II 



EDUCATIONAL INDICATORS DEFINED AND CLASSIFIED 



What is an Indicator ? 

Webster's Collegiate Dictionary defines an indicator 
as "one that shows or points out; an indication or sign," 
This same definition also serves as a global description of 
educational indicators . In other words, educational 
indicators should show or point out instances where our 
educational programs are doing what they were designed to 
do. Also, educational indicators should provide an indi- 
cation or a sign that the quality which was built into 
the educational program actually is there. An additional 
point must be emphasized: All educational indicators are 
quanti fiablef that is, they are described by some numerical 
value. 

The- numeric quantity obtained for an indicator must be 
assigned some meaning and then be compared against some 
acceptable standard. For example, a pupil-teacher ratio of 
25:1 usually is considered to be educationally more desir- 
able than one of 30:1. The lower ratio tends to indicate 
thai: each student could receive more personal assistance 
from the teacher. An indicator which might suggest the 
availability of economic resources for a given school dis- 
trict would be better if a higher rather than a lower 
numerical value is obtained. Similarly, high scores would 
be desirable for the indicator student test results, while 
low scores would be preferable for the indicator incidence 
of drug abuse by young adults. 

The crucial point to keep in mind is that when you 
decide to employ an educational indicator, you should define 
for yourself the magnitude which represents acceptable 
achievement BEFORE you can make any judgment about your 
actual indicator findings. 

The Oregon Department of Education has defined an 
indicator as "a descriptor in quantifiable terms, of the 
status at a specified point in time of a significant 



condition or variable which provides evidence useful for an 
analysis of progress toward a goal or objective."^ Note the 
Oregon definition is both precise and comprehensive. Oregon 
also identifies the three important elements that appear in 
the definition: 

1. The expression is quantifiable — data does exist 
or can be collected to show "how much" of the 
indicator exists. 

2. The condition or variable that is described 
has, by general agreement, a relationship to 
the goal with which it is associated. 

3. The measurement is associated with a point 
in time (Ibid., p. 3). 

Oregon illustrates an indicator that contains these three 
elements with this example: 

The number of high sch ool seniors who did 

^ 

volunteer work in a community social agency 

w — — 

during the school year (Ibid., p. 3) 

^3) 

(Such an indicator probably would be used in support of an 
objective in a senior social studies course.) 

Whether the reader prefers the dictionary or the Oregon 
approach to defining indicators, the same result will ensue. 
Indicators will help you evaluate the various educational 
programs in your school. Once you have settled on a 
definition of educational indicators, you need to know the 
kinds of indicators which may be available to you. 

Classification of Educational Indicators 

Educational indicators can be obtained from a wide 
variety of sources depending upon your purpose in using them. 
Functionally, indicators can be classified as: 

Administrative — including population and census 
data, finances, sources of revenue, etc. 



•••Robert Clemmer, Dwight Fairbanks, Mary Hall, James Iiapara, and 
Charles Nelson, Indicators and Statewide Assessment (Denver: 
Cooperative Accountability Project 1974), p. 3. Subsequent quotations 
froir. this source are cited in parentheses following each quotation. 
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School Program — including pupil-teacher ratio, 
student intelligence and achievement informa- 
tion, dropout rate, success after graduation, 
etc. 

Planning — including population trends, impaction 
of minorities, availability of local economic 
resources, etc. 

Each school district probably will wish to design a 
classification system which best fits its organizational 
structure. States employing educational indicators have 
chosen differing patterns for classifying them. Several 
examples are summarized in the following paragraphs. 

OREGON 

Oregon has classified educational indicators according 
to their uses under four major categories: (1) input, (2) 
context, (3) performance, and (4) societal* 

1. An input indicator describes a condition or 

variable over which the school has some control 
and which affects the school's ability to achieve 
an instructional, management, or support goal. 
(Ibid., p. 3) 

Input indicators tend to be concerned with the basic 
"raw material" of the learning process. Examples include 
number and sex of students; number, sex, and educational 
background of teachers; pupil- teacher ratio; quantity of 
facilities or equipment; quality of facilities or equipment; 
number of teacher aides and support personnel; number of 
counselors and administrative personnel; etc. 

Let's see how you might use an input indicator as part 
of the educational decision-making process. We'll say, for 
the sake of example, that your pupil-teacher ratio is 32:1 
in elementary school A. For the past three years grade 3 
students at school A have been achieving four months lower 
than equivalent students at school B which has a pupil- 
teacher ratio of 24:1. In all likelihood you would use this 
input indicator data as justification for providing addi- 
tional teachers and facilities at school A to lower the 
pupil-teacher ratio to around 25:1. 

Another use of input indicators might involve an over- 
crowding problem. Let's say that one section of your city 
is rapidly expanding in terms of new home construction. You 
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have an existing school in that particular area built to 
accommodate 1,200 elementary youngsters. Presently you have 
1,400 students enrolled in the school, and city planning 
projections estimate that two years from now, 2,800 students 
will be living in the area in ques.tion. Other schools in 
the district are functioning at 110% student capacity. What 
do you do? First, the school board may float a bond to 
build a new school which will accommodate 2,000 students in 
the overcrowded area of the city. But since the new school 
cannot be operational for two years or more, obviously 
educational administrators and planners will have to devise 
some stopgap enrollment pattern for the existing school 
until the new one can be occupied. Some alternative such 
as split sessions, extended day, or year-round schooling 
will need to be employed. Also, more administrators, 
teachers, aides, and support personnel will need to be hired. 

2. A context indicator describes a condition or 
variable over which the school has little or 
no control. It affects the school's ability 
to achieve an instructional, management, or 
support goal. (Ibid. , p. 3) 

Context indicators describe the learning environment for 
a given school or school district. The key thing to keep in 
mind when dealing with context indicators is that they tend 
to be "givens" you just have to live with. Examples of con- 
text indicators include socioeconomic background of students; 
ethnic background of students; ability of students; avail- 
ability of economic resources; operating costs by budget 
category, grades K-12; average daily membership; operating 
costs per resident average daily membership; number of 
recipients for free and reduced school lunch program; number 
of children in families below the poverty level; etc. 

While, for the most part, context indicators cannot be 
changed, they often provide sufficient justification for a 
school or school district to qualify for additional funds 
under various state and federal categorical programs. School 
districts in close proximity to large Department of Defense 
installations, for example, qualify fcr Public Law 874 funds. 
Similarly, school districts having a high incidence of 
minority children qualify for federal funds under Title I of 
the Elementary and Secondary Education Act. Many states 
also have funds available for the special education child. 
If a district has a number of youngsters who qualify for 
special education programs, it can apply for and receive 
state financial assistance. 
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For any given school district, one mill of taxes will 
yield only a giver sum of tax dollars. This sum usually is 
large for urban centers; smaller for suburban "bedroom 
communities"; and smallest for rural communities. The taxes 
per mill indicator is a crucial ingredient for state legis- 
lation for the support of public education when a "power 
equalization" formula is employed. So, while you can't do 
too much about changing context indicators, you certainly 
can use these indicators prudently to secure additional 
funds for your schools. 

3. A performance indicator describes a measurable 
or observable behavior or variable used to 
determine program effectiveness or efficiency. 
Data may concern: (a) student performance 
scores, or (b) a program variable such as 
instructional process or availability of 
learning experiences. (Ibid., p. 3) 

Performance indicators will be the most frequently used 
indicators of success of Educational programs. Examples 
include student test results; anecdotal records of student 
behavior; number of students completing graduation require- 
ments; number of courses using individualized instruction; 
number of courses offered; number of students who enroll in 
two- or four-year institutions upon graduation; number of 
learning situations outside of the school that are avu^l- 
able to students; number and ethnic background of dropouts; 
number of students enrolled in vocational education courses; 
number of young adults (ages 18-21) who register and do 
vote in local elections; number of graduates who are work- 
ing in the occupation they trained for one, three, and five 
years after high school graduation; percentage of high 
school graduates who continued their education in state 
institutions; etc. 

Based upon the goals and objectives of a school dis- 
trict's educational program, a series of performance 
indicators could be assigned to key objectives. The 
performance indicators then would provide additional 
information to student test results to determine the degree 
to which students are achieving the desired outcomes of 
learning. Performance indicators not only can measure the 
cognitive skills usually associated with academic learning, 
but they also can measure the affect or attitudes which 
students have about the learning process. 

Follow-up studies on graduates might tend to show that 
some required courses are rated as having "no value" after 
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graduation. Armed with this information! administrators and 
school board officials may consider giving the required 
course or courses in question more practical utility or, if 
this is not feasible, the course (s) might be dropped from 
the "required" list. 

Test results also might show that junior high school 
students are below the state or national average in social 
studies, for example* The teaching staff then can determine 
whether the social studies questions on the test represented 
a sample of the objectives of the junior high social studies 
program. If the test does reflect the objectives of the 
social studies program, then the program may need to be 
strengthened in certain content areas. If the test does not 
reflect the objectives of the social studies program, then 
another test which does should be administered before any 
decision is made concerning possible change of course content. 
Thus performance indicators can provide continuous informa- 
tion concerning change in the educational programs of a 
school district. 

4. A societal indicator describes a measurable 
aspect of a social condition affected to 
some degree by education. (Ibid., p. 3) 

Societal indicators provide some inkling of the value 
which schools h=ave fostered or failed to foster in their 
students. Examples include the number of arrests for 
delinquency; suicide rate for young adults; incidence of 
drug abuse by young adults; incidence of alcoholism by 
young adults; employment rate of recent graduates; number 
of juvenile, school case, misdemeanor problems handled by 
school district; marriage rate of graduates; divorce rate 
of graduates; incidence of venereal disease by young adults; 
number of illegitimate births; number of persons 65 years 
of age and over in the district population; number of wel- 
fare recipients among young adults, etc. 

For the sake of illustra !:ion, suppose the number of 
young adult arrests for alcoholism has been on the increase 
in your school for the last three years. Let's also say 
that your school district's alcohol and drug abuse portion 
of a required high school health course has been operational 
for the last seven years. The indicator of increased 
alcoholism tends to suggest that the instructional strategy 
or methodology employed could be improved. If the rate of 
alcoholism continues to increase, perhaps a more realistic 
and positive approach could be stressed in your high school 
health course. 
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Take another example of a societal indicator. Let's 
say that the number of persons 65 years of age and over in 
your school district has stabilized. A query of the senior 
citizens reveals their interest in enrolling in arts and 
crafts courses as part of the school district's community 
school program. For those districts in which fiscal policy 
allows all the public to benefit, teachers could be hired 
and the school building made available for senior citizens' 
arts and crafts courses. For those districts where policy 
precludes financing courses for senior citizens, perhaps one 
of the senior citizens could serve as an arts and crafts 
teacher, and the district could make a school building 
available for the senior citizens' class during after-school 
hours when custodians normally would be working. 

In summarizing the Oregon classification of educational 
indicators, it is apparent the classification is both compre- 
hensive and precise. In their suggestions for using 
indicators, Oregon educators suggest that indicators should 
be (1) related to agreed upon goals; (2) derived from 
reliable and valid data? (3) derived from data that will 
continue to be collected so that comparisons over time may 
be made; and (4) derived from data for which the measurement 
techniques have stability over time. (Ibid., p. 3) 

NEW YORK 

In New York the State Education Department has developed 
a Performance Indicators in Education (PIE) program for local 
school districts to use in evaluating their educational pro- 
grams. The classification of educational indicators under 
the PIE program is as follows: 

1. Input factors (stuJ.ent characteristics at the 
start of the evaluation) (Ibid., p. 12) Similar 

to the student portion of Oregon's input indicators. 

2. Process factors (Ibid., p. 12) Characteristics 
of the educational program, especially the 
methodology employed to achieve student learning. 

3. Output factors (student characteristics at the 
conclusion of the evaluation) (Ibid., p. 12) 
Similar to the performance indicators under the 
Oregon classification. 

4. Surrounding conditions (community characteristics) 
(Ibid., p. 12) Similar to the context indicators 
under the Oregon classification. 
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The New York classification is an interesting one for 
at least two reasons. First, the focus is on evaluation. 
The classification lends itself to the classical research 
design usually found in educational research. Subjects are 
selected for the study (input indicators) . Some methodolog- 
ical procedures are employed in the study {process indicators) , 
Results are gathered and analyzed (output indicators) . The 
results are interpreted within the constraints over which the 
experimenter has little or no control {surrounding conditions 
indicators) . As the reader will understand mdre fully later » 
the New York emphasis on evaluation represents greater 
potential for positive change than does Oregon's approach 
which is designed to be carried out at the measurement (or 
assessment) level. 

Second, the PIE program is designed so each school dis- 
trict may develop its own indicator profile which can be 
compared with the profiles of other school districts having 
similar community characteristics. This analysis provides 
local educational decision makers with a series of profile 
comparisons which may assist them in the selection of the 
alternative decision which holds the most promise for success 
in their district. 

The New York State Education Department administrators 
of the PIE program describe its purpose as follows: "To 
estimate the difference between (a) tks> level of output which 
could be expected if the schools' contribution to output were 
not significant , and (b) the actual level of the schools' 
output. The difference between the two values is taken as 
an indicator of the schools' performance," (Ibid., p. 12) 

What this statement really says is that a school dis- 
trict can predict the expected performance students in each 
school can attain. PIE administrators do this by using 
statistical equations which assign different weights to 
socioeconomic status, student ability, sex (girls usually 
achieve more than boys at the lower grade levels) , etc. for 
each child in each school; this refers to part (a) of the 
quoted statement of purpose. The actual level of each 
student's performance is compared with the performance level 
that was predicted for the child. These "difference values" 
then are tabulated for each grade in each school. When the 
actual values for the school as a whole exceed the predicted 
values, this is taken as an indicator that the school has 
positively influenced its learners, and vice versa; here is 
reference to the part (b) of the quoted purpose. 



PENNSYLVANIA 

''Che Pennsylvania use of indicators is an extension of 
the statewide assessment program which has been going on in 
that state for the past three or four years. In fact, the 
already described Oregon classification is an extension of 
what has been accomplished in Pennsylvania. Heretofore, 
Pennsylvania has chosen to use the expressions conditions or 
variables in lieu of the expression educational indicators , 
Each condition or variable is used either singly or in combi- 
nation to predict student performance. More than 50 variables 
have been classified into the following three categories: 

1. Characterisvics of students, (Ibid., p. 12) (These 
overlap into the input, context; and performance 
indicators in the Oregon classification.) 

2. Characteristics of teachers, (Ibid., p. 12) (These 
are the same as Oregon's input indicators as re- 
lated to teachers.) 

3. School and community characteristics, (Ibid., p. 12) 
(These are the same as input and context indicators 
in the Oregon classification.) 

The purpose of the Pennsylvania educators is to determine 
the exact relevance of each of these conditions to student 
performance in relation to each of ten educational or life 
goals. For example, five conditions hcive been singled out 
to predict performance of fifth grade students in relation 
to the goal of achieving self -understanding , These conditions are 
"father's occupation; housing conditions in the community; 
teacher stability; teacher experience; and school subsidy 
per student." (Ibid., p. 12) It can easily be seen that 
Pennsylvania, through its use of conditions or variables, is 
trying to find what range of factors influence the academic 
and attitudinal performance of its students. 

Since it is one of the participating states in the 
Cooperative Accountability Project, Pennsylvania is preparing 
materials (which will be published by CAP) emphasizing the 
state's educational quality assessment findings and discuss- 
ing these (indicators) as they relate to goals of education 
in Pennsylvania. When these findings are available, it would 
be reasonable to assxame that the correlates of educational 
performance uncovered will have general application for school 
districts across the nation. 
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Summary 



The purpose of this chapter has been to present a series 
of alternative ways for classifying educational indicators. 
Each reader will need to choose the system or combination of 
classification systems to employ at a particular local school 
level. The dictionary or functional approach, when used with 
the four purpose questions described in Chapter I, can pro- 
vide a rather simplistic v/ay for using indicators in the 
evaluation of educational programs. The Oregon classification 
is quite comprehensive and rigorous. It also introduces the 
notion of elevating societal factors to the same status of 
input, context, and performance educational indicators. The 
New York classification follows the classical experimental 
design usually found in educational research. To think that 
one can use educational indicators without conducting some 
form of study or research is to be kidding oneself. The New 
York approach should provide local educators with insights 
concerning how to use and follow up on indicators from year 
to year. The Pennsylvania approach to educational indicators 
provides a concentrated approach to handling performance 
indicators. The statistical precision employed in the 
Pennsylvania approach should provide local district educators 
with an "already validated" means for dealing with educational 
indicator data. The composite perspective of these three 
state classif icatory systems will give each local educator 
an opportunity to put his school district's policy for 
handling educational indicators in a more comprehensive 
framework. 

Now that the reader knows what educational indicators 
are and how they can be classified, we're ready to find out 
how to obtain the educational indicators you may wish to use. 
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CHAPTER III 



WHERE DO YOU FIND EDUCATIONAL INDICATORS? 



Assigning Responsibility 

When a local school district decides it wishes to use 
educational indicator information as part of the process of 
evaluating its educational programs, it will be necessary to 
assign the responsibility for the work (and it is work) to 
some member of the district administrative staff. The staff 
member selected should possess, as a minimum, a background 
in testing and measurement, psychological statistics, and 
educational research design. Depending on local circum- 
stances, the person should be assigned to the indicator task 
on a full or part-time basis. 

In a large school district a testing department or a 
research department {or both) may already exist and can 
assume the responsibility for gatheririg and using educational 
indicator information. / 

Medium and small-sized school districts usually do not 
hdve research or testing departments, so either the head 
counselor or head of the mathematics department might 
logically be assigned the indicator responsibility. If this 
additional assignment would prove to be an excessive burden 
(and in many cases it would), then the individual's counsel- 
ing or teaching schedule would need to be reduced. 

After a district coordinator for indicators has been 
found at the district level, he or she may request that an 
indicator contact person be appointed for each school in the 
district. This contact person then could supervise the , 
collection of indicator data at the school building level and 
provide input to the district coordinator for indicators. 

Local school boards may wish to appoint an advisory 
committee for educational indicators composed of adminis- 
trators, teachers, students, parents, and other taxpayers. 
The advisory committee should work with the district 
indicators coordinator to assure that a comprehensive mix of 
indicator data is made available to the school board for 



decision-making purposes. Some state accountability laws re- 
quire local districts to appoint an advisory committee for 
accountability purposes. Should this be the case in your 
district, by all means assign the indicator advisory 
responsibility to the accountability committee. If your 
district does not have an accountability committee, you 
should examine the various district "standing committees" 
to see if one of them can take on the educational indicators 
responsibility before you elect to appoint a special advisory 
group for educational indicators. There is a tendency in 
public education to overextend the number of committees 
advising the board of education. The suggestion for an edu- 
cational indicators advisory committee is in no way a 
recommendation to contribute to this proliferation. Yet it 
should be stressed that committee assistance , especially in 
the early planning stages, can hp.lp to assure that an adequat 
number and variety of indicators will be collected for your 
analysis purposes. 

Where Do You Locate Existing Indicator Data ? 

The question of where to begin gathering indicator data 
is far from superficial. Even readers who are familiar with 
indicators may find the question less than easy to answer 
because of the limited scoj-'e of their exprjience with educa- 
tional indicators. For those readers now being introduced 
to indicators for the first time, the question is of obvious 
initial concern. 

Existing indicator data are available at the following 
sources: 



Source 



Indicator Information 



1. 



Local School or Dis- 
trict Central 
Administration 



1. School census; ethnic dis- 
tribution in city and 
schools; operational school 
budget; total school budget; 
average daily attendance; 
grade distributions by 
course, grade level, and 
school; number of students 
by grade and sex; number of 
certified teachers; percent- 
age of administrative and 
teacher turnover; results 
of educational research 
studies conducted within 
the district; number of 
special education students 
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Source 



Indicator Information 



2. City or County 
Government 



and prograns; number of 
gifted stud^mts and programs 
(usually ignored) ; student 
test results; dropout data; 
parent and community needs 
assessment data; state and 
federal categorical aid pro- 
grams; school lunch program, 
transportation routes and 
costs; school mill levy and 
other educational planning 
factors. 

2. Data on property and use 
taxes ; demographic informa- 
tion which shows area^ of 
population decrease, 
stability, or increase; 
welfare information; home 
building projections; 
industrial and commercial 
wealth of the area, along 
with projections for the 
future; employment and un- 
employment data; health 
data including the incidence 
of venereal disease; number 
of adults over 25 who do not 
have a high school diploma; 
number of people over 16 who 
have had no formal years of 
school; park and recreation 
data; number of work permits 
issued to minors; crime 
statistics including 
juvenile offenses; incidence 
of alcohol and drug abuse; 
vital statistics including 
births, deaths, marriages, 
divorces, aid to dependent 
children; assessed valua- 
tion data by region; and 
planning studies showing 
decreased, stable, or in- 
creased projections for the 
short- and long-term future. 
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Source 



Indicator Information 



State Government 



a) State Department 
of Education 



3. Data available at the dis- 
trict and county level also 
are provided for all coun- 
ties in the state. Addi- 
tionally, the various 
agencies of state govern- 
ment also collect and 
distribute other data on 
request. 

a) Ethnic distribution of 
students; number of 
segregated schools; 
nuraber of racial and 
ethnic group school dis- 
trict personnel; number 
of schools with a high 
concentration of low- 
income students; number 
of teacher aides and 
certificated teachers 
by position and level; 
operating costs by budget 
category. Grades 1-12, 
including per pupil costs 
private and parochial 
school enrollment: grade 
levels, number of adminis- 
trators and teachftrs, 
high school graduates; 
proj*^cted enrollment by 
grade; number of public 
high school graduates by 
age and sex; number of 
students enrolled in 
vocational education 
courses; number of 
recipients for free and 
reduced school lunch pro- 
gram; community college 
enrollment in vocational 
courses; number of school 
bus accidents: type, date 
and time; number of 
counselors: percentage 
of assigned counseling 
time; number of children 



Source 



Indicator Infomation 



b) Department of State 



c) Department of Health 



d) Department of Social 
Services 



in public kindergartens; 
number of students and 
programs for special 
education; compendium of 
current laws related to 
public education. 

b) Number of registered 
voters; percentage of 
eligible registered 
voters; number voting in 
an election; number of 
eligible voters; number 

of businesses: established 
during the year, total 
number y bankruptcies; etc* 

c) Number of cases and rates 
of venereal disease; 
number of: live births, 
illegitimate births, and 
infant deaths; number of 
health professionals 
needed in tne state's 
communities, by profession; 
number of persons receiving 
out-patient and in-patient 
care in county mental 
health hospitals; number 

of hospitals, patient care 
facilities, and profession- 
al staff in the state; 
incidence of alcohol and 
drug abuse in the state; 
number of people in various 
health occupations; niimber 
of persons 65 years of age 
and over; and the total 
population. 

d) Number of persons partic 
ipating in abundant food 
and food stamp programs; 
nximber of welfare 
recipients; number of 
unemployed and under- 
utilized disadvantaged 
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Source 



Indicator Information 



persons averaged over 12 
months; total number of 
children served by public 
and voluntary child wel- 
fare agencies and 
institutions; number of 
persons receiving general 
assistance aid payments; 
number of persons re- 
ceiving aid to dependent 
children payments (non- 
medical) . 



e) Department of Law 
(Attorney General's 
Office) 



f) Department of Revenue 



e) Number of arrests: 
offense, age (under 18, 
18 and over) ; crimes re- 
ported last year, last 
three years, and projec- 
tion for next year; 
number of juvenile 
school case, misdemeanor 
problems handled; number 
of juvenile delinquency 
commitments to state 
institutions; number of 
consumer problems handled 
by Legal Aid: sales con- 
tracts, garnishments, 
wage claims, bankruptcies, 
other; number of family 
problems handled by Legal 
Aid: divorce and annul- 
ment, separation, non- 
support, custody and 
guardianship, paternity, 
adoption, other. 

f) Data on property, use, 
county, city, and state 
taxes; number of state 
income tax returns filed 
by county and city; 
number of elderly persons 
with no taxable income 

or less than $3,000 of 
taxable income; distribu- 
tion of wealth around the 
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ERIC 



Source 



Indicator Information 



g) Department of 
Transportation 



state; number of dollars 
in payrolls, by industry. 

g) Number of young adult 
(under 24) drivers in- 
volved in accidents; 
number of schools with 
organized safety educa- 
tion programs, driver 
education, student 
accident reporting, and 
transportation safety 
programs; number of 
schools with adult driver 
education including 
problem driver courses, 
refresher courses, and 
special driver courses 
for handicapped; number 
of schools with programs 
related to alcohol in- 
volvement in traffic 
accidents; number of 
traffic violation con- 
victions for young adults; 
percentage of licensed 
drivers under 25 and 
25-34 years of age with 
percentage involved in 
all accidents, and percent- 
age in fatal accidents; 
number of vehicle deaths 
per 100,000 miles driven; 
percentage of male drivers 
and percentage of female 
drivers by number of 
accidents and number of 
violations; nxamber of 
driving accidents by age 
and residence. ♦ 



4. Federal Government 



4, Virtually all data available 
at the state level also is 
available from some agency 
of the federal government 
which will enable you to 
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ERIC 



Source 



Indicator Information 



a) U.S. Department of 
Commerce, Bureau of 
Census 



compare your local or state 
information on a regional 
or national basis. This is 
especially true for the 
Department of Health, Educa- 
tion, and W«=»lfarc. 

a) Percentage of adults 

(age 25 years and older) 
with an eighth grade 
education or less; number 
of adults without a high 
school education (age 25 
years and older) ? number 
of adults (age 25 and 
older) with 0-4 years of 
education; total number 
of persons below the 
poverty level- 



Other Sources 



There is a wide variety 
of other sources of 
educational indicator 
data; these include, 
but are not limited to: 
professional organiza- 
tions; business and 
commerce; state 
colleges and univer- 
sities; etc. 

a) Professional Organi- 
zations (Council of 
Chief State School 
Officers, National 
Education Association, 
American Federation 
of Teachers, American 
Educational Research 
Association, state 
and local professional 
associations , etc . ) 



a) Reports of professional 
organizations on educa- 
tional studies often re- 
veal data which can prove 
relevant for educational 
indicators. These 
indicators cover the 
complete range of educa- 
tion programs and concerns 
of local school districts. 



Source 



Indicator Information 



Business and Commerce 
(Insurance companies, 
oil companies, banks, 
utilities, etc.) 



b) National business and 
commercial firms tend to 
conduct studies of inter- 
est to public education 
and also publish materials 
as a public service. The 
studies and materials of 
the following are most 
notable : 

Insurance Companies ; 
Health and mortality in- 
formation, especially 
medical and dental health 
and, more recently, 
information regarding 
alcohol and drug abuse. 

Oil Companies ; Driver 
education, env i r onmen t a 1 
education, and conserva- 
tion materials and data. 

Banks : Materials and data 
on tne value of money 
usually presented through 
some kind of junior bank- 
ing program? national, 
regional, state, and 
local data in the form 
of consumer price index, 
cost of living index, 
and similar economic 
indicators. 

Utilities ; Materials and 
data relevant to science 
and communication usually 
are made available to the 
public schools. Results 
of research similarly are 
presented through the 
medium of published re- 
ports or articles in 
professional journals. 
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Source 



Indicator Information 



c) State Colleges and 
Universities 
(especially their 
bureaus of educational, 
business, or commerce 
research) 



c) Number of state high 

school graduates enrolled 
in freshman remedial 
courses in English and 
mathematics; student grades 
in college, by course; 
assessed valuation of 
property; expenditures of 
local government (includ- 
ing local schools) ; amount 
of personal income: tax, 
average income, average 
tax; population by various 
age ranges; number of 
children in families with 
various income levels: 
high, average, below 
poverty level; number of 
people living in urban or 
rural conditions; niunber 
of people by land area, 
density; number of business 
failures; number of dollars 
in payrolls, by industry. 



As the reader readily can see from the sample listings, 
indicators covering a wide range of educational programs and 
purposes already are being collected by agencies at the local, 
county, state, and federal government levels as well as by 
other sources. A host of educational indicators are awaiting 
your use. 

You might compare the selection of educational indicators 
to the use of audio-visual materials. Many audio-visuals 
already exist. But you must be selective in their use if they 
are to be effective in furthering instructional efforts. You 
must ask yourself, "What is the most efficient and effective 

way for our students to learn ?" Let the nature of the 

educational objective dictate the teacher methodology and 
support materials (possibly audio-visual) to be used, not vice 
versa. And so it is when you look at the broad educational 
program goals and objectives for your school district. 

In other words, you don't go to the Instructional Materials 
Center in your district, preview a film, and say, "For what 
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class can this film be used?" The same holds true for exist- 
ing educational indicators for evaluation of local educa- 
tional programs. To help you determine which indicators can 
be of viable use to you, ask i'ourself these questions: 

- What are the crucial things our students must learn 
from the objectives of the (grade program? 

- What evidence do we need to say that our students 

have not only learned , but are practicing it 

in their behavior? 

- What individual or series of indicator data are 
already being collected which could answer the 
question, "Have our students learned and are they 
practicing ?" 

When the last question has been answered, you probably have 
associated the appropriate indicators with the desired 
objectives to be evaluated. 

How Do You Generate Educational Indicators ? 

For many educational program evaluations, educational 
indicators may already exist; but if your district is like 
most, there will be some crucial educational objectives for 
which no data are being gathered. There will come a time when 
all the existing indicators from local, county, state, federal, 
and other sources will not satisfy your district's needs. When 
this happens, you will find yourself in the educational re- 
search business. 

After searching for already-existing educational indica- 
tors, you may find you either have a complete void or an 
incomplete number of indicators for one or more educational 
objectives. A statement of indicator requirements remaining 
unfilled then must be developed. Based upon the total state- 
ment of requirements, the district coordinator for indicators 
should list a series of alternate ways for gathering the 
needed information. These questions should be explored: 

- Will student test results provide the necessary 
information? 

- Are student, teacher, or parent questionnaires 
required? 

- Do you need to measure the attitudes and opinions 
of a sampling of people in the community to solve 
your problem? 



- Can you solicit a local, county, or state agency to 
get the required data? 

- Can you use unavailable indicator data as the basis 
for a proposal for state or federal categorical 
funds to do research in uncovered areas? 

When the district coordinator for indicators has angered 
these and other related questions, he or she will be in a 
position to move into the instrument development and data 
gathering phases of Iccal educational research. 

The instrument development phase of educational research 
is both a tedious and time-consuming exercise. (This is why 
it wes recommended earlier in this chapter that the district 
coordinator for indicators should have a background in 
educational tests and measurements.) Initial decisions which 
must be made include: 

- ^'Tho is to be sampled? 

- What form shall the measuring instrument assume? 

- Do you need to construct a questionnaire? If so, 
will the same version of the questionnaire be 
appropriate for each group of people to be sampled 
(e.g., students, teachers, parents, etc.)? 

- Is the interview technique appropriate? If so, will 
an interview guide need to be developed so that each 
interviewer asks the same questions in sequence? 

Regardless of whether questionnaires, interviews, or some 
other means of data collection are to be utilized, the next 
step for the district coordinator for indicators is to develop 
a set of operational definitions. In psychology and in 
education an operational definition defines the specific array 
of behaviors that describe the term being defined. For 
example, if you have an indicator in your study called student 
adherence to democratic principles in everyday life, it might 
be operationally defined as follows: 

A student will be said to be adhering to democratic 
principles? in his everyday life if, and only if, he 
is: tolerant of another person's different ethnic 
background and values; accepts fellow students as 
equals regardless of their intellectual prowess; 
willingly accepts the majority decision even if it 
conflicts with his own; willing to do his share in 



helping the family situation along? willing to 
treat co-workers as coequals on the job; and 
willing to accept responsibility for the con- 
sequences of his actions* 

If the above were an operational definition for your dis- 
trict, the district coordinator for indicators would seek tc 
provide information relevant to one or more of the phrases of 
the definition until all the phrases are adequately sampled 
in the data gathering processes. The coordinator then would 
have to ask, "Do I have enough items to measure this indicator?" 
and "When is enough really enough?" There are no exact rules 
for answering these questions because indicators will vary 
in complexity, but a good rule of thumb might be: not less 
than three nor more than five questions should relate to each 
phrase of an operational definition. Of course, depending 
upon the complexity of the indicator, you niay wish to vary 
the rule of thumb. 

If you are going to utilize questionnaires or interview 
guides, you may wish to try them on a small number of people 
before you actually gather your data on the full sample for 
your study. A pilot study will assist you in determining the 
validity and reliability of your questionnaire or interview 
guide. Validity and reliability are two characteristics of 
a good measuring instrument which will enable you to have 
trust in the data you will receive in the larger sampling 
for your study. a test is valid if it measures what it is 
intended to measure and nothing else. 

In some cases the district coordinator for indicators 
will be primarily interested in content validity. In other 
words, do the responses made in the pilot study tell you all 
you need to know about what it is you're trying to measure? 
Your coordinator may find that everyone in the pilot study 
answers one or more questions exactly the same way. When this 
occurs it may be decided to accept this information and 
eliminate these particular questions from the larger sampling. 

Reliability has to do with consistent results. In this 
regard your district coordinator for indicators may wish to 
exercise an option. It is possible to statistically tabulate 
a reliability coefficient based upon internal consistency OR 
it is possible to include the pilot study sample in the 
major sampling and correlate the pilot study and "real study" 
results. One of the options should be utilized. (Remember 
it was noted that the district coordinator for indicators 
should have experience in psychological and educational 
statistics? You can see the need for these competencies.) 



After the pilot study comes the major study. Tools of 
the study must be readied, distributed, completed, returned; 
then the district coordinator for indicators must tally the 
results. The coordinator may wish to aggregate the results 
by computing the mean (arithmetic average) for each aspect 
of the data obtained. When this is done, you have an edu- 
cational indicator based on data not available from other 
sources. Sound easy? It can be time consuming, but it's 
not really terribly difficult. 

Summary 

In this chapter it has been suggested that a district 
coordinator for indicators be named. It also was suggested 
that school building contact persons and an advisory 
committee be called upon to assist the district coordinator. 
A sampling of already-available educdtional indicators was 
provided and, while it is not a complete list, it should 
activate the reader's imagination to the point of asking, 
"Where are indicators already available in my school dis- 
trict,- county, state, federal agencies, and elsewhere?" 
Some elementary procedures also were suggested for generating 
your own educational indicator data. Now. . .what do you do 
with the educational indicator data after you get it? 
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CHAPTER IV 



HOW DO YOU GROUP EDUCATIONAL INDICATOR DATA? 



The process of grouping educational indicator data is 
a relatively easy, mechanical operation. Nonetheless it is 
crucial that the process be performed and be performed 
correctly. 

Prioritizing Objectives and Indicators 

A priority rating should be assigned to each educational 
objective which is to be evaluated in a given educational 
program. Of course some objectives are more crucial than 
others. Therefore educational objectives may be classified 
as very important, important, and nice to know. Educational 
objectives also can be classified as discriminative or growth. 
A discriminative objective is one which the bright students 
will learn well, while the less intellectually endowed 
students might not learn it as well. 'Test items on achieve- 
ment tests that "purport to measure comprehensive applications 
or extensions of basic principles usually are assessing 
discriminative objectives. On the other hand, growth objec^ 
i-iv^s are those objectives written ag?.inst the major building 
blocks of a course or against the learning concepts which 
are required fcr further learning in the course. It is 
desirable that all or virtually all students (say 80%) achieve 
growth objectives. 

To begin the prioritizing process, first you classify 
your objectives as discriminative or growth. Second, you make 
a decision as to whether each objective is very important, 
important, or nice to know. At this point you are ready to 
prioritize your educational indicators that you have related 
to each educational objective. Similarly, educational 
indicators identified as being relevant to a given educational 
objective also can be classified as very important, important, 
or nice to know. 

When you have classified the priority of all the educa- 
tional indicators for each educational objective, you then 



can see the degree to which each indicator or set of indica- 
tors should yield valid results. This relationship is shown 
in Figure 1: 



FIGURE 1 

DIAGRAM OF OBJECTIVES AND 
EDUCATIONAL INDICATORS CLASSIFICATION 



EDUCATIONAL 
OBJECTIVE 



CLASSIFICATION 

Is the Objective ; Is the Indicator : 

^^^Very Important? 
Very Important?^^^ Important? 

^"■^Nice to Know? 



Very Important? 

Important?^ Important? 

•Nice to Know? 



-Very Important? 

Nice to Know? cdr Important? 

-Nice to Know? 



As Figure 1 suggests, each educational objective should 
be classified and then, in turn, each educational indicator 
should be classified and related to the objective. When all 
educational indicators for a given objective have been 
classified according to importance (assuming more than one 
indicator per objective) , the district coordinator for 
indicators then can assign a priority to the data the combined 
indicators should yield. This process results in the prior- 
itizing of objectives and their corresponding educational 
indicators . 

Another way of illustrating the combination of classi- 
fying and prioritizing is to present various hypothetical 
educational objectives and indicator classifications for given 
objectives, and then indicate the confidence a district 
coordinator for indicators snould have in the resulting data. 
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Let's assume for a given objective that three educational 
indicators have been identified. The range of objective and 
indicator classifications, together with the district 
coordinator's decision concerning confidence in a hypothet- 
ical example, are contained in Figure 2: 



FIGURE 2 

COMBINATION OF WAYS OF CLASSXFYING OBJECTIVES AND 
INDICATORS AND THE CONFIDENCE ASSIGNED TO EACH COMBINATION 
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Figure 2 Continued 
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Figure 2 Continued 
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Figure 2 Continued 
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It is readily apparent that prioritizing has to do with 
placing a value on something. The third column of Figure 2 
suggests a series of confidence levels which a district 
coordinator for indicators might assign on the basis of 
indicator data. 

The district coordinator for indicators also will want 
to establish some criterion standards of success for each 
indicator to be used. In other words, how large must the 
magnitude of a given educational indicator be for the coordi- 
nator to say, "Yes, this indicator says we're making it," or 
"No, this indicator says we've got problems"? 

It should be noted there is no particular rule of thumb 
for establishing criterion standards of success. The district 
coordinator for indicators will have to establish cutoff 
points on the educational indicator scale and define a mean- 
ing for each point identified. Note, however, that it is 
crucial to set criterion standards of success for each 
indicator BEFORE the magnitude of the indicator Is known. 

For example, let's say you're evaluating a math program. 
A test is being administered to a class in October. The test 
yields three math scores you will use as educational indicators 
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math facts, math processes, and computational skill. Based 
on the October test, the grade placement to be expected for 
the average child will be 5.1 (this method of averaging will 
be explained in greater detail in Chapter V) . Your dis- 
trict coordinator for indicators may establish the following 
standards for each of the three scores: 

Average Student Achievement Criterion Interpretation 

4.8 or less Our program needs improvement 

4.9 - 5.3 Our program is "making it" 
5.4 or more Our program is exemplary 

Again it must be emphasized that setting criterion 
standards of success and their corresponding interpretations 
must occur befors the educational indicator is measured. Then 
when the educational indicator measurement has been made, the 
magnitude which results already has a meaning for interpreta- 
tion. 

Summary 

This chapter has suggested ways in which the district 
coordinator for indicators can sort out the indicators that 
go with each educational objective being evaluated. Both 
educational objectives and educational indicators should be 
classified as very important, important, or nice to know. When 
the coordinator "lays out" the classified objectives and the 
classified indicators, some judgment can be made about the 
confidence the indicator information should offer. It also 
has been suggested that, while going through the classifica- 
tion process, the coordinator should establish cutoff points 
aloig each indicator. These cutoff points then are assigned 
interpretations. The latter process usually is referred to 
as establishing criterion standards. The crucial aspect of 
everything presented in this chapter is that the whole process 
of classifying and prioritizing must occur before the educa- 
tional indicators are quantified. 

At this point you've discovered indicators ... found out 
something about the kinds of indicators which exist. . .classi- 
fied indicators. . .grouped and related them to educational 
objectives. Your next question may well be, "So what? 
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CHAPTER V 



HOW DO YOU INTERPRET INDICATOR INFORMATION? 



Evaluation 

Educational evaluation is judgment based on criteria. 
This classic definition has withstood the test of more than 
a half-century of use in the field of public education. 
Educational evaluation is a terminal activity associated with 
educational measurement or assessment. Evaluation seeks to 
answer the question, "So what?" regarding drt-.a. 

Measurement is the art of quantifying something. The 
purpose of educational measurement is to assess whether the 
student has attained (learned) the various objectives of the 
curricula. For all practical purposes in education, the 
words measurement and assessment are synonymous. (The largest 
organization dedicated to assessment. National Assessment of 
Educational Progress or NAEP, tends to reaffirm this relation- 
ship between measurement and assessment. NAEP reports merely 
present the data that have been gathered; they make no specific 
recommendations as to how learning deficiencies should be 
ameliorated. ) 

On the other hand, when you evaluate a given program and 
find components lacking, it is a natural tendency to seek 
alternate strategies to correct problems. So it should be 
with educational indicators. If you're not willing to try to 
improve your educational programs by suggesting alternatives 
to problems, then you should forget the whole assessment or 
measurement process. You're wasting your time and other 
people's money when you do not follow through. If you're not 
willing to face reality and be agents for educational change 
when change is warranted, then don't collect educational 
indicator data. It will only embarrass you and make you feel 
uneasy. 

Now, for those of you who are still reading and have 
decided to evaluate educational indicators — welcome. Educa- 
tional evaluation can be both interesting and intriguing 
(although when you are identified as an evaluator, some 
educators tend to treat you as if you have an infectious 
disease) . 



The New York and Pennsylvania programs described 
earlier treat indicators from a local school district evalu- 
ative frame of reference. Oregon, while a leader in the 
educational indicators arena, has chosen to operate at the 
assessment level only. 

Evaluation Climate 

If you're wise, you will lay a foundation of mutual 
trust and regard before you expect to be able to function at 
the evaluation level with educational programs. People are 
people. People can do and do do some funny things. You also 
should remember that administrators and teachers are people 
too. They feel, and have a right to feel, that the educa- 
tional programs they administer carry the skill and 
imagination which they have built into the art of teaching. 
Then you come along and say, "We can do this better." It is 
essential that you make such a suggestion in an objective 
and emotionally neutral way. 

Educators tend to react in similar manner to managers 
in industry when* confronted with evaluation. Some managers 
are delighted to have another person come in and take a 
fresh look at the operation they are charged with. These 
same managers tend to be open to suggestions for change and 
do not necessarily feel threatened by the idea that the job 
could be accomplished in a better way. Other managers are 
inclined to regard evaluation in poker player terms — with 
"cards close to the vest." The latter managers seem to 
resist any suggestions for change since they feel that, if 
they admit the job could be done better, they also are 
admitting they're really not good administrators. Unfortun- 
ately, some highly talented people still regard their station 
in life as tentative and precarious. In a school district 
of any size, you are going to have to deal with administrators 
and teachers who run the entire gamut of the security- 
insecurity continuum. 

Kow do you establish an objective evaluative environment 
"^t the local school district level? Perhaps the first thing 
you should do is meet with the superintendent and request the 
board of education to issue a policy statement on educational 
evaluation. The exact wording will vary from district to 
district, but the policy statement should contain the follow- 
ing thoughts : 

- The members of the school board and the administration 

of district are committed to quality education 

for our children. 
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- The members of the school board feel they are very 
fortunate to have secured the services of truly 
outstanding and innovative educators for both the 
administrative and teaching staffs. 

- In an attempt to continually increase the quality 
of the educational program offerings in the dis- 
trict, a rigorous evaluation of each program will be 
undertaken in a three-year cycle. 

- Programs will be evaluated during the first year of 
the three-year cycle, and suggested improvements will 
be doc\imented. 

- During the second year of the cycle the programs will 
be modified to include suggested improvements. 

- During the third year of the three-year cycle the 
modified curriculum will be operational in the class- 
room. 

- During the fourth year (first year of a new three- 
year cycle) the educational program will be evaluated 
again. 

- It is the intention of the members of the school 
board and the superintendent that the educational 
resources of both the administrative and teaching 
staffs' will be fully utilized to modify our 
curriculum. 

- To lend support to this policy of quality education 
for all our school children, the board is earmarking 

$ to defray the expense of evaluators, 

curriculum specialists, and $ for sub- 
stitute teachers and administrators in order to 
release our personnel to work on curriculum revisions. 

Such a policy statement would tend to create a non- 
threatening atmosphere in the school district, provide a 
climate for objective evaluation, and demonstrate that the 
board of education truly is "behind the evaluation program." 
In such an atmosphere, the evaluators might even find some 
teacher or administrator who would, in a friendly way, offer 
to buy them a cup of coffee. 

Interpretation of Results 



The district coordinator for educational indicators will 
find that indicator data can be treated like any other data 



for reporting purposes. The nature of the educational 
indicator data will similarly be of the nominal, ordinal, 
and interval variety. The coordinator will tend to use the 
same statistical routines on educational indicator data that 
would be used for routine educational data. 

Some readers may not be familiar with educational 
statistics, while it is not the intent of this monograph 
to deal with statistics, some mention needs to be made con- 
cerning norm-referenced and criterion-referenced testing. 
A norm-referenced test is developed by a commercial testing 
company and has national norms. When a student takes this 
type of test, his achievement can be compared with equivalent 
students on either a national or regional basis. A criterion- 
referenced test is a locally produced test designed to measure 
the student mastery of learning objectives contained in a 
given school district's program. Criterion-referenced tests 
also are available commercially. Th*5se test items, together 
with statements of the educational objectives they are de- 
signed to measure, can be purchased by local school districts. 
If the commercially available criterion-referenced objectives 
are identical to the local school district's objectives, the 
test can be administered as purchased. When the district and 
commercial criterion-referenced objectives are not in 
consonance, the criterion test writing task must rest with 
the local school district. 

(Up to this point in the monograph, the author has 
stressed the importance of reletting indicators to objectives 
of school programs. This emphasis might tend to put the 
author in the criterion-referenced testing camp. I don't 
mind. I actually have friends who are criterion-referenced 
testing protagonists.) 

A point should be made, however, that the comparison of 
data yielded from each measurement device (norm vs. criterion) 
need not necessarily result in an either-or type of measure- 
ment decision. Certainly a given norm-referenced instrument 
does not contain items which comprehensively measure the 
content of a particular district's English, math, reading, 
or social studies programs, but chances are that it does 
measure a significant portion of these respective programs 
(or else why did the faculty recommend the use of a given 
norm-referenced test??) . The questions on the norm-referenced 
test can be compared to the school district's program objec- 
tives; when this is done, some local objectives may be 
reflected by five or more questions, others by one or two 
questions, and still others will have no related questions 
on the norm-referenced instrument. Such an analysis will 



thus reveal objectives which have been partially measured 
and others that have not been measured at all. This analysis 
then becomes the basis for constructing a criterion-referenced 
instrument. 

Now, at testing time, both instruments are administered. 
The usual norm data are available for use in the traditional 
manner for each student. Similarly the student answers for 
both instruments are grouped by ..ocal school district educa- 
tional objectives to ascertain whether or not the obtained 
ease indices (percentage of studen^- who answer a test item 
correctly) meet the established criteria. This dual approach 
enables local educators to conduct their district-wide assess- 
ment and relate both types of data to the effectiveness of 
their local educational programs. 

Many school districts which employ norm-referenced 
measurement also purport to have programs which individualize 
instruction. Certain individualized programs such as 
Individually Prescribed Instruction (IPI) , Individually 
Guided Education (IGE) , Project PLAN, etc. each have the 
notion of individual student learning as their goals. With 
such programs, norm-referenced tests can be employed to 
determine whether a given student is working up to his or her 
potential through comparison of actual achievement with some 
measure of expected achievement. 

In the early 1960s the California Testing Bureau (now a 
division of McGraw-Hill) incorporated an intellectual status 
index into the California Achievement Test Battery. The 
intellectual status index employed the mental age portion of 
the student's I.Q. to project how much progress a given 
child should make in a school year. The technical conversion 
grade equivalent measures for each I.Q. then are made. The 
practice was to compare actual achievement with expected 
achievement and determine whether the student was an under- 
achiever , achiever , or overachiever on a given subtest. One 
problem experienced when using the intellectual status index 
was that this technique tended to overestimate a given 
student's predicted achievement. 

In 1969 Myklebust developed a learning quotient (L.Q.) 
which is said to be a more reliable estimate of expected 
performance. 2 The learning quotient includes a more definitive 



2h. E. Myklebust, editor, Prog^ss in Learning Disabil 
ities, Vol. I (New York, Greene Stratton, 1969) . 



and comprehensive index of expectancy as it takes into account 
not only the mental age but also the chronological age, re- 
presenting physiological maturity, and the grade age, repre- 
senting an index of school experience. A learning quotient 
can be computed for each area of achievement (each subtest) , 
thus providing individualistic information for each child. 

Actual achievement (scores earned on achievement tests) 
then is related to this expectancy and multiplied by 100, 
resulting in a learning quotient: 

Achievement Age x 100 . ^ . . . 
Expectancy Age = Learning Quotient 

Learning quotients of 89 and below are interpreted to 
represent substantial discrepancy between actual and expected 
achievement, indicating a student learning deficiency requir- 
ing special attention. Learning quotients of roughly 90 to 
93 represent problem areas still requiring attention but not 
quite as severe as the 89 and below L.Q.s. Learning quotients 
of roughly 94 and above indicate that a child is achieving 
at a level commensurate with his expectancy. A relative 
pattern of strengths and weaknesses becomes apparent as L.Q.s 
in different achievement areas ere compared. 

Whether the local district employs a combination of 
norm and criterion-referenced data to assess their educa- 
tional programs on a local basis or embarks on a computerized 
approach to measure the individualized learning of its 
students, the results come out the same. Areas of program 
strengths and areas of improvement result. It remains for 
the educational leaders in the district to evaluate these data 
and initiate programs of change, where appropriate. 

Prediction or Expectancy 

The educational indicators you use in evaluating your 
educational program might be used to predict the future level 
of some component of the school district's program. For 
example, let's say that a medium-sized school district 
(enrollment 36,000+) is located in close proximity to a large 
military installation. The city is listed as one of the top 
ten fastest growing communities in the United States. Let's 
say that the 1970-71 enrollment was 31,809; the 1971-72 
enrollment was 33,124; the 1972-73 enrollment was 34,467. 
What should be the enrollment for the 1973-74 school year? 
The data are shown in Table I: 
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TABLE I 

PREDICTING STUDENT ENROLLMENT 




1970-71 1971-72 1972-73 



1973-74 
(?) 



By connecting the 1970-71, 1971-72, and 1972-73 enroll- 
ment data, a straight line (or line of best fit) is obtained. 
By extending the line until it intersects a vertical line 
extending upward from the 1973-74 location on the horizontal 
axis, the intersection is slightly higher than 36,000 students. 

But (to continue our example) when school started in 
September, 1973, there were only 35,200 children enrolled. 
Where are the other 800 students? Well, during the 1972-73 
school year, the Department of Defense personnel budget was 
reduced, and the local military installation was cut 380 man- 
power positions for the new fiscal year (beginning July 1) • 
The military commander decided to take his personnel reduc- 
tion through normal attrition (not getting replacements for 
380 personnel whose tour of duty or enlistment would terminate 
by June 30) . If the school planners did not take this addi- 
tional information into account (@2.1 children/family), the 
September enrollment was overestimated by 800 students. If 
the cost/student expenditure in that district was $900/student, 
this error would have amounted to $720,000 in the budget. Also 
on the pupil-teacher ratio of 24:1, an additional 33 teachers 
would have been hired. If the average cost/ teacher was $10,300 
then $339,900 would have been spent unnecessarily on teachers 
whose services really were not needed. To compound the felony, 
the school district would receive less state aid than it had 
anticipated. Also, the Public Law 874 funds would be decreased 

The point being made in this illustration is that you not 
only must be sensitive to educational indicators, but also must 
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be "on top of" any other factors which could influence edu- 
cational predictions. School budgets are built on educational 
indicator data, but the planner must be willing to make 
adjustments whenever some influencing factor varies. 

Based on student input factors for each class, every 
class of students becomes a rather unique entity. The 
students' academic potential can be predicted. When actual 
performance is less than expected, the logical question is, 
"What happened?" Educational indicator infonriation often can 
help to interpret the basic achievement data in terms of the 
"What happened?" query. 

Many tests are scored on the basis of a grade equivalent 
score. Grade equivalent scores divide the year into ten 
months so that 5.0 refers to September in the fifth grade, 
5.1 is October in the fifth grade, and so forth. When using 
tests that report results in grade eq\\ivalent scores, it is 
essential that the obtained results be compared to the actual 
month in which the test was administered. An average grade 
equivalent score of 5.5 for a test administered in February 
places the local student achievement right on the national 
norm. To interpret this score, the district coordinator for 
indicators would have to see if the students who took the 
test in question were within the normal range of intelligence 
(90-110). If they are, the district coordinator may say, 
"Our students are progressing well in terms of other similar 
fifth grade students in the nation." If the average I.Q. of 
the students who took the test was 95, the district coordina- 
tor can say, "Our students are exceeding the performance of 
similar fifth grade students in the nation." If the average 
I.Q. of the students who took the test was 112, the district 
coordinator would have to say, "Our students are not achiev- 
ing as well as similar fifth grade students across the nation. 
We need to look closely at what our students are learning." 

Oregon is using a format for building a matrix showing 
the direct relationship between goals, concepts, and appro- 
priate indicators. Oregon has adopted goals for education 
in the state which are related to six life roles. These are 
learner, producer, citizen, consumer, individual, and family 
member. This matrix format is shown in Table II. 

A district coordinator for indicators might find it handy 
to add one or two columns to the Oregon matrix. In these 
additional coltimns could be placed the indicator values and 
the year these data were gathered. This would be helpful 
especially for planning and for interpretation of results. 
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One last point. Earlier in the monograph the statement 
was made that indicators should have at least one of four 
purposes for being used. Those four purposes bear repeating. 
Indicators should tell you: 

1. How are our students doing in school? 

2. How do our students fare after they leave school? 

3. How can this information be of assistance to the 
instructional staff? 

4. Are we getting as much as we should for our 
school dollar? 

As you relate educational indicator data to educational 
objectives, keep these four questions in mind. Your inter- 
pretation of indicator data will be better utilized if it is 
related to these four major purposes. 



48 



CO 
Eh 

O 

2n 

OS 
O 
QO 

uo 
»^ 

w CO 

CO ^ 

o o 

b 

w 

(3Q CO Eh 

I O Eh 
Eh CO 

go 

gco 
o 

Hi 04 

o 

Q 

H 0. 
Eh 

CO Z 
N 
O 
O 
D 
CO 



V4 

u 

•H 



<4 

4- 

Q) 
•H 

U 
O 

CO 



8 

•H 

0 4J 

« g 

CM 

1 c 

\o (d 



u 
u 



0 Q) 
•H >i O 
4-> O U4 
(d H 

01 § ^ 

o c 13 



























u 








Q) 


0 








H 




% 




o 


•H 


U) 


u 




4J 


O 


U) 






(d 




Q) 






u 




c 


2 










04 




•a 










m 








M 






0 










•H 




Q) 






4J 




U 






(d 






1 


0^ 


u 




(d 




Q) 


0 










H 








(d 


04 




0 


M-l 


u 






M-t 


o 




Q) 








H 




• 


Q) 


u 


(d 












Q) 


o 






u 


Q) 








Q) 










04 


(d 






2 


U) 


0 











4J 




(d 




•H 








O4 Q) 




0 U 




U -H 


U 


P4 0 




04 x: 


0 


(d u 


a 










C 0) 




-H 0) 




^ U 




3 ''J 




S 0 



u 

U 

I 

V4 

CU 



(0 



a 4J 





4-) 

•o (d 

<P 4-) 

0) (0 

4-) O 

m % 

4J 



•s 

4-» 

o 

4-» 

c 

0) 

o 
u 



o 

4-> 

0) 
H 

•3 

0) 

(d 

4-» 
&4 4J 



I 

Q) 
.C 
U 

cd 

0) 

o 
(d 

iH 

(d 
4) 

o 

4-> 

>i 
4J 

•§5 




i 

(0 




I 



en 



Q) 

•a 

0) 

«d 
<u 
Co 



49 



ERIC 



Summary 



In Chapter V a case has been m,\de for establishing a 
positive climate for evaluation at the school district level. 
Discussion also focused on forms of testing used to gather 
performance indicator data. Don't be forced into either the 
norm-referenced or criterion-referenced camps. Use the best 
parts of both, and apologize to no one. Some suggestions and 
cautions have been mentioned dealing with interpretations 
when attempting predictions. The Oregon matrix format has 
much to recommend it. Finally, the importance of interpret- 
ing indicator data in terms of four purpose questions again 
was stressed. 
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CHAPTER VI 

DISSEMINATING EDUCATIONAL INDICATOR REPORTS 



Reports generated by indicator evaluation must be 
written in different forms so that different target audiences 
can read and understand them. A technical report replete 
with statistical comparisons can be developed for educational 
researchers and key educational administrators, A "plain 
English" version of the report should be developed for 
parents, students, and the public so that laymen can read 
and react to the findings. 

The outline for each version of an indicator evaluation 
report should follow the taxonomy (classification structure) 
for educational decision makers described in a monograph this 
author assisted in writing for the National Association of 
State Boards of Education entitled statewide Educational 
Evaluation . 

Indicator evaluation data should provide partial answers 
to these questions: 

1. What should our school graduates know; be able to 
do; believe? What do they now know? 

2. To what extent should the schools ameliorate class 
differences; emphasize individual excellence; 
insist on common minimum standards of performance, 
treat the exceptional student exceptionally? 

3. What do our citizens believe should be the goals 
of education; consider the essential priorities 
of education? feel about the value of student 
preferences and needs? 

4. What should we do about the contradictions affect- 
ing education between freedom, progress, and 



^D. Tronsgaurd, M. Grady, and E. D. Coon, Statewide Educational 
Evaluation (Denver: National Association of State Boards of Education, 
1974) . 



necessity; between goals of equality and goals 
of excellence? between what we believe and 
what we are doing; between societal and 
individual needs and preferences? 

If indicator evaluation reports respond to this taxonomy 
of educational questions, the achievements, the successes, 
the areas for improvement, and the follow-up activities come 
quickly and sharply into focus. Indi.r itor reports should 
provide the locus for innovation antl aprovement in public 
education, and these reports should allow the local board of 
education members to receive hard data so they can make more 
valid educational decisions for the future. 

Using educational indicators for evaluative purposes 
will add a comprehensive dimension to your school program. 
The use of indicators can help to provide to teachers the 
opportunities to implement the new, innovative techniques they 
have been reading and learning about. 

An underlying commandment should pervade the collection, 
interpretation, and reporting V7hich emanates from the use of 
educational indicators. This commandment is: 

Thou Shalt not use educational indicators as an excuse 
to conduct a witch-hunt or to search for a scapegoat , 

Getting along with one's neighbors is an exercise in 
human relations. So too is keeping faith and peace in the 
educational communxty. As you conduct your work in all phases 
dealing with educational indicators, treat other people pro- 
fessionally, and they'll treat you that way too. 
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CAP PUBLICATIONS 

Report 
Number 

1 Annotated Bibliography of the State Educational Accountability Repository. 
Phyiiis Hawthorne. Revised April 1974. Wisconsin. 

2 Legislation by the States: Accountability and Assessment in Education. Phyllis 
Hawthorne. Revised August 1974. Wisconsin. 

4 State Goals for Elementary and Secondary Education. Susan Ketchum Ribble. 
Revised September 1973. Wisconsin. ERIC ED 083747. 

5 Characteristics of and Proposed Models for State Accountability Legislation. 
Phyllis Hawthorne and Archie A. Buchmiller. April 1973. Wisconsin. ERIC ED 
078 514 

6 Accountability: A Bibliography. Gordon P. Hanson. July 1973. Wisconsin. 
ERIC ED 084630. 

20 Keeping the Public Informed: Accent on Accountability. A digest of the 
Michigan dissemination model. Erwin P. Bettinghaus and Gerald R. Miller. 
1973. Michigan. 

21 A Dissemination System for State Accountability Programs. Erwin P. 
Bettinghaus and Gerald R. Miller. June 1973. Michigan. 

22 Developing a Large Scale Assessment Program. Frank B. Womer. July 1973. 
Minnesota. 

23 Indicators and Statewide Assessment. March 1974. Oregon. 

24 Fioles of the Participants in Educational Accountability. Carl E. Wilsey and 
Glenn B. Schroeder. 1974. Colorado. 

25 Using Educational Indicators for Program Accountability. Michael J. Grady, Jr. 
September 1 974. Colorado. 



NOTE Documents with ERIC reference numbers can be obtained through the usual ERIC 
procedures 



Copies may be obtained from: 

STATE EDUCATIONAL ACCOUNTABILITY REPOSITORY 
WISCONSIN DEPARTMENT 0^ PUBLIC INSTRUCTION 
126 LANGDON STREET 
MADISON, WISCONSIN 53702 
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